What is a factor variable?

Factors are used for

1. categorical variables with a fixed and known set of possible values.

• E.g., day_born = Sunday, Monday, Tuesday, …, Saturday

2. displaying character vectors in non-alphabetical order.

Creating a Factor – Base R

word_data <- tibble(word = c("apple", "dog", "banana", "cat",
                             "banana", "Queen Elizabeth", "dog")
           )
word_data

# A tibble: 7 × 1
  word           
  <chr>          
1 apple          
2 dog            
3 banana         
4 cat            
5 banana         
6 Queen Elizabeth
7 dog            

factor(word_data$word)

[1] apple           dog             banana          cat            
[5] banana          Queen Elizabeth dog            
Levels: apple banana cat dog Queen Elizabeth

Creating a Factor – Base R

When you create a factor variable from a vector…

• Every unique element in the vector becomes a level.
• The levels are ordered alphabetically.
• The elements are no longer displayed in quotes.

Creating a Factor – Base R

You can specify the order of the levels with the level argument.

factor(word_data$word, levels = c("banana","apple","cat",
                             "dog", "Queen Elizabeth")
       )

[1] apple           dog             banana          cat            
[5] banana          Queen Elizabeth dog            
Levels: banana apple cat dog Queen Elizabeth

forcats

Creating a Factor – forcats

word_data <- tibble(word = c("apple", "dog", "banana", "cat",
                             "banana", "Queen Elizabeth", "dog")
           )
word_data |> 
  pull(word)

[1] "apple"           "dog"             "banana"          "cat"            
[5] "banana"          "Queen Elizabeth" "dog"            

word_data |> 
  mutate(word = fct(x = word)) |> 
  pull(word)

[1] apple           dog             banana          cat            
[5] banana          Queen Elizabeth dog            
Levels: apple dog banana cat Queen Elizabeth

• With forcats, the levels are automatically ordered in the order of first appearance.

Creating a Factor – forcats

You can still specify the order of the levels with the level argument.

word_data |> 
  mutate(word = fct(x = word, 
                    levels = c("apple", "banana", "cat",
                               "dog", "Queen Elizabeth")
                    )
         ) |> 
  pull(word)

[1] apple           dog             banana          cat            
[5] banana          Queen Elizabeth dog            
Levels: apple banana cat dog Queen Elizabeth

You can also specify non-present levels.

word_data |>
  mutate(word = fct(x = word, 
                    levels = c("apple", "banana", "cat", "dog",
                               "Queen Elizabeth", "King Charles")
                    )
         ) |> 
  pull(word)

[1] apple           dog             banana          cat            
[5] banana          Queen Elizabeth dog            
Levels: apple banana cat dog Queen Elizabeth King Charles

Re-coding a Factor – fct_recode

Change the existing levels with the syntax:

<new level> = <old level>.

word_data |> 
  mutate(word = fct_recode(.f = word,
                           "fruit" = "apple",
                           "fruit" = "banana",
                           "pet"   = "cat",
                           "pet"   = "dog")
         ) |> 
  pull(word)

[1] fruit           pet             fruit           pet            
[5] fruit           Queen Elizabeth pet            
Levels: fruit pet Queen Elizabeth

Collapsing a Factor – fct_collapse

Collapse the existing levels with the syntax:

<new level> = c(<old levels>).

word_data |> 
  mutate(word = fct_collapse(.f = word,
                           "fruit" = c("apple", "banana"),
                           "pet"   = c("cat", "dog")
                           )
         ) |> 
  pull(word)

[1] fruit           pet             fruit           pet            
[5] fruit           Queen Elizabeth pet            
Levels: fruit pet Queen Elizabeth

Re-leveling a Factor –fct_relevel

Change the order of the levels of an existing factor.

word_data |>
  pull(word) |> 
  levels()

[1] "fruit"           "pet"             "Queen Elizabeth"

word_data |> 
  mutate(word = fct_relevel(.f = word,
                            c("Queen Elizabeth", "pet", "fruit")
                            )
         ) |> 
  pull(word) |> 
  levels()

[1] "Queen Elizabeth" "pet"             "fruit"          

Unspecified levels will be in the same order at the end of the list.

word_data |> 
  mutate(word = fct_relevel(.f = word, 
                            "pet")
         ) |> 
  pull(word) |> 
  levels()

[1] "pet"             "fruit"           "Queen Elizabeth"

Re-coding a Factor – fct_recode

library(liver)
data(cereal)
str(cereal$manuf)

 Factor w/ 7 levels "A","G","K","N",..: 4 6 3 3 7 2 3 2 7 5 ...

cereal |> 
  mutate(manuf = case_when(manuf == "A" ~ "American Home Food Products", 
                           manuf == "G" ~ "General Mills", 
                           manuf == "K" ~ "Kelloggs", 
                           manuf == "N" ~ "Nabisco", 
                           manuf == "P" ~ "Post", 
                           manuf == "Q" ~ "Quaker Oats", 
                           manuf == "R" ~ "Ralston Purina"),
         manuf = as.factor(manuf)) |> 
  select(manuf) |> 
  summary()

                         manuf   
 American Home Food Products: 1  
 General Mills              :22  
 Kelloggs                   :23  
 Nabisco                    : 6  
 Post                       : 9  
 Quaker Oats                : 8  
 Ralston Purina             : 8  

cereal |> 
  mutate(manuf = fct_recode(manuf, 
                             "American Home Food Products" = "A", 
                             "General Mills" = "G", 
                             "Kelloggs" = "K", 
                             "Nabisco" = "N", 
                             "Post" = "P", 
                             "Quaker Oats" = "Q", 
                             "Ralston Purina" = "R")) |> 
  select(manuf) |> 
  summary()

                         manuf   
 American Home Food Products: 1  
 General Mills              :22  
 Kelloggs                   :23  
 Nabisco                    : 6  
 Post                       : 9  
 Quaker Oats                : 8  
 Ralston Purina             : 8  

Re-ordering Factors in ggplot2

library(ggridges)
cereal_recode |> 
  ggplot(aes(x = sugars, 
             y = manuf, 
             fill = manuf)) +
  geom_density_ridges() +
  theme_minimal() +
  theme(legend.position = "none")

cereal_recode |> 
  ggplot(aes(x = sugars, 
             y = fct_reorder(.f = manuf, 
                             .x = sugars,
                             .fun = mean), 
             fill = manuf)) +
  geom_density_ridges() +
  theme_minimal() +
  theme(legend.position = "none")

Re-ordering Factors in ggplot2

cereal_recode |> 
  group_by(manuf, shelf) |> 
  summarise(avg_sugar = mean(sugars, na.rm = TRUE)) |> 
  ggplot(aes(x = shelf, 
             y = avg_sugar, 
             color = manuf)) +
  geom_line(size = 1.5) +
  theme_minimal() +
  labs(color = "Manuf.")

cereal_recode |> 
  group_by(manuf, shelf) |> 
  summarise(avg_sugar = mean(sugars, na.rm = TRUE)) |> 
  ggplot(aes(x = shelf, 
             y = avg_sugar, 
             color = fct_reorder2(manuf, .x = shelf, .y = avg_sugar))) +
  geom_line(size = 1.5) +
  theme_minimal() +
  labs(color = "Manuf.")

Glamour of Graphics

• See Will Chase’s 2020 RStudio Conference Presentation - Glamour of Graphics

Tidy Data

Tidy data…

• is rectangular.
• has observations as rows and variables as columns.
• has different formats for different tasks.

R4DS

Consequences of Messy Data

Illustration by Allison Horst

• Tidy: use the same tools in similar ways for different datasets.
• Messy: create unique tools that are difficult to generalize.

Creating Tidy Data

We may need to transform our data to turn it into the version of tidy that is best for a task at hand.

Illustration by Allison Horst

Creating Tidy Data

We want to look at mean cereal nutrients based on shelf.

• The data are in a wide format – a separate column for each nutrient.
• Transforming the data will make plotting easier.

library(liver)
data(cereal)
head(cereal)

Creating Tidy Data

cereal_wide <- cereal |> 
  group_by(shelf) |> 
  summarise(across(calories:vitamins, mean))

my_colors <- c("calories_col" = "steelblue", "sugars_col" = "orange3")

cereal_wide |> 
  ggplot() +
  geom_point(aes(x = shelf, y = calories, color = "calories_col")) +
  geom_line(aes(x = shelf, y = calories, color = "calories_col")) + 
  geom_point(aes(x = shelf, y = sugars, color = "sugars_col")) +
  geom_line(aes(x = shelf, y = sugars, color = "sugars_col")) +
  scale_color_manual(values = my_colors, labels = names(my_colors)) +
  labs(x = "Shelf", y = "", subtitle = "Mean Amount", color = "Nutrient")

cereal_long<- cereal |> 
  pivot_longer(cols = calories:vitamins,
               names_to = "Nutrient",
               values_to = "Amount") |> 
  group_by(shelf, Nutrient) |> 
  summarise(mean_amount = mean(Amount))

cereal_long |> 
  ggplot(aes(x = shelf, 
             y = mean_amount, 
             color = Nutrient)) +
  geom_point() +
  geom_line() +
  labs(x = "Shelf", y = "", subtitle = "Mean Amount")

Manual Method

Consider daily rainfall observed in SLO in January 2023.

• The data is in a human-friendly form (like a calendar).
• Each week has a row, and each day has a column.

Data source

How would you manually convert this to long format?

Manual Method: Steps

Computational Approach

We can use pivot_longer() to turn a wide dataset into a long(er) dataset.

pivot_longer()

Take a wide dataset and turn it into a long daaset.

• cols – specify the columns that should be pivoted.
  • Do not include the names of ID columns (columns to not be pivoted).
• names_to – the name of the new column containing the old column names.
• values_to – the name of the new column containing the old column values.

pivot_longer()

library(readxl)
slo_rainfall <- read_xlsx("data/2023-rainfall-slo.xlsx")

slo_rainfall |> 
  mutate(across(Sunday:Saturday, as.numeric)) |> 
  pivot_longer(cols      = Sunday:Saturday,
               names_to  = "Day_of_Week",
               values_to = "Daily_Rainfall")

pivot_wider()

Take a long dataset and turn it into a wide daaset.

• id_cols – specify the column(s) that contain the ID for unique rows in the wide dataset.
• names_from – the name of the column containing the new column names.
• values_from – the name of the column containing the new column values.

pivot_wider()

We calculate the mean amount of protein for cereals on each shelf and for each manufacturer.

mean_protein <- cereal |> 
  group_by(manuf, shelf) |> 
  summarize(mean_protein = mean(protein))

pivot_wider()

mean_protein |> 
  arrange(shelf) |> 
  pivot_wider(id_cols = manuf,
              names_from = shelf,
              values_from = mean_protein)

Better names in pivot_wider()

mean_protein |> 
  arrange(shelf) |> 
  pivot_wider(id_cols = manuf,
              names_from = shelf,
              values_from = mean_protein,
              names_prefix = "Shelf_")

Relational Data

Multiple, interconnected tables of data are called relational.

• It is the relation between data sets, not just the individual data sets themselves, that are important.

IMDb movie relational data

Data Joins

We can combine (join) data tables based on their relations.

Keys

A key uniquely identifies an observation in a data set.

• To combine (join) two datasets, the key needs to be present in both.

inner_join()

Keeps obsertvations when their keys are present in both datasets.

inner_join(): IMDb Example

directors_genres

movies_directors

inner_join(directors_genres, movies_directors)

inner_join(): IMDb Example

What if our key does not have the same name?

directors_genres

directors

inner_join(directors_genres, 
           directors, 
           by = join_by(director_id == id))

Piping Joins

Remember: the dataset you pipe in becomes the first argument of the function you are piping into!

• So if you are using a pipe, you will only be specifying the right dataset inside the join function.

inner_join(directors_genres, movies_directors)

…is equivalent to…

directors_genres |> 
  inner_join(movies_directors)

Mutating Joins

• left_join() – keep only (and all) observations in the left data set

• right_join() – keep only (and all) observations in the right data set

• full_join() – keep all observations in both data sets

Mutating Joins

Which directors would remain for each of the following?

directors_genres |> 
  distinct(director_id)

movies_directors |> 
  distinct(director_id)

Filtering Joins: semi_join()

Keeps observations when their keys are present in both datasets, but only keeps variables from the first dataset.

Filtering Joins: semi_join()

directors_genres |> 
  semi_join(movies_directors)

directors_genres |>
  filter(director_id %in% movies_directors$director_id)

Filtering Joins: anti_join()

Removes observations when their keys are present in both datasets, and only keeps variables from the first dataset.

Filtering Joins: anti_join()

directors_genres |> 
  anti_join(movies_directors)

directors_genres |>
  filter(!director_id %in% movies_directors$director_id)

PA 4: Military Spending

Today you will be tidying messy data to explore the relationship between countries of the world and military spending.

To do…

• PA 4: Military Spending
  • Due Thursday, 2/1 at 8:00am

Thursday, February 2

Today we will…

• Review Lab 4: Teacher Evaluations
• Miscellaneous…
  • Clean Variable Names
  • Lifecycle Stages
• Extensions to Relational Data
• Lab 4: Childcare Costs

Lab 3: Teacher Evaluations

• The tidyverse package automatically loads ggplot2, dplyr, readr, etc. – do not load these twice!

• Do not use mutate() + distinct(), when you can use summarize()!

mutate() vs summarise()

Game Plan: Rate My Professor

Clean Variable Names with janitor

Data from external sources likely has variable names not ideally formatted for R.

Names may…

• contain spaces.
• start with numbers.
• start with a mix of capital and lower case letters.

names(military)[1:12]

 [1] "Country"        "Notes"          "Reporting year" "1988"          
 [5] "1989"           "1990"           "1991"           "1992"          
 [9] "1993"           "1994"           "1995"           "1996"          

Clean Variable Names with janitor

The janitor package converts all variable names in a dataset to snake_case.

Names will…

• start with a lower case letter.
• have spaces filled in with _.

library(janitor)
military_clean_names <- military |> 
  clean_names()

names(military_clean_names)[1:12]

 [1] "country"        "notes"          "reporting_year" "x1988"         
 [5] "x1989"          "x1990"          "x1991"          "x1992"         
 [9] "x1993"          "x1994"          "x1995"          "x1996"         

Lifceycle Stages

As packages get updated, the functions and function arguments included in those packages will change.

• The accepted syntax for a function may change.
• A function/functionality may disappear.

Learn more about lifecycle stages of packages, functions, function arguments in R.

Deprecated Functions

A deprecated functionality has a better alternative available and is scheduled for removal.

• You get a warning telling you what to use instead.

military_clean |> 
  filter(across(Notes:`2019`, is.na)) |> 
  slice_head(n = 3) |> 
  select(1:8)

Warning: Using `across()` in `filter()` was deprecated in dplyr 1.0.8.
ℹ Please use `if_any()` or `if_all()` instead.

# A tibble: 3 × 8
  Country      Notes `Reporting year` `1988` `1989` `1990` `1991` `1992`
  <chr>        <chr> <chr>            <chr>  <chr>  <chr>  <chr>  <chr> 
1 Africa       <NA>  <NA>             <NA>   <NA>   <NA>   <NA>   <NA>  
2 North Africa <NA>  <NA>             <NA>   <NA>   <NA>   <NA>   <NA>  
3 Sub-Saharan  <NA>  <NA>             <NA>   <NA>   <NA>   <NA>   <NA>  

Deprecated Functions

You should not use deprecated functions!

Instead, we use…

military_clean |>
  filter(if_all(Notes:`2019`, ~ is.na(.x))) |> 
  slice_head(n = 3) |> 
  select(1:8)

# A tibble: 3 × 8
  Country      Notes `Reporting year` `1988` `1989` `1990` `1991` `1992`
  <chr>        <chr> <chr>            <chr>  <chr>  <chr>  <chr>  <chr> 
1 Africa       <NA>  <NA>             <NA>   <NA>   <NA>   <NA>   <NA>  
2 North Africa <NA>  <NA>             <NA>   <NA>   <NA>   <NA>   <NA>  
3 Sub-Saharan  <NA>  <NA>             <NA>   <NA>   <NA>   <NA>   <NA>  

Superceded Functions

A superseded functionality has a better alternative, but is not going away .

• This is a softer alternative to deprecation.
• A superseded function will not give a warning (since there’s no risk if you keep using it), but the documentation will give you a recommendation.

Relational Data

When we work with multiple tables of data, we say we are working with relational data.

• It is the relations, not just the individual datasets, that are important.

When we work with relational data, we rely on keys.

• A key uniquely identifies an observation in a dataset.
• A key allows us to relate datasets to each other

IMDb Movies Data

How can we find each director’s active years?

Joining Multiple Data Sets

directors[1:4,]

# A tibble: 4 × 3
     id first_name last_name
  <dbl> <chr>      <chr>    
1   429 Andrew     Adamson  
2  2931 Darren     Aronofsky
3  9247 Zach       Braff    
4 11652 James (I)  Cameron  

movies_directors[1:4,]

# A tibble: 4 × 2
  director_id movie_id
        <dbl>    <dbl>
1         429   300229
2        2931   254943
3        9247   124110
4       11652    10920

movies[1:4,]

# A tibble: 4 × 4
     id name           year  rank
  <dbl> <chr>         <dbl> <dbl>
1 10920 Aliens         1986  8.20
2 17173 Animal House   1978  7.5 
3 18979 Apollo 13      1995  7.5 
4 30959 Batman Begins  2005 NA   

movies_directors |> 
  inner_join(directors, 
             join_by(director_id == id))

movies_directors |> 
  inner_join(directors, 
             join_by(director_id == id)) |> 
  inner_join(movies,
             join_by(movie_id == id)) |> 
  rename(movie_name = name)

Joining on Multiple Variables

Consider the rodent data from Lab 2.

• We want to add species_id to the rodent measurements.

species

measurements

species |> 
  full_join(measurements,
            join_by(species == species, 
                    genus == genus_name))

Lab 4: Teacher Evaluations

Filtering Joins

semi_join(): Keeps values found in another data set

anti_join(): Keeps values not found in another data set

Workflow

1. Load packages + read in data.

library(tidyverse)
data_original <- read_csv(file = "path/to/data.csv")

2. Clean data and save changes – use this data from now on!

data_clean <- data_original |> 
  mutate(across(x1:x5, ~ as.factor(.x)),
         new_var = if_else(...))

3. Create subsets from your cleaned data for specific tasks.

demographics_subj <- data_clean |> 
  distinct(subj, keep_all = TRUE)

4. Output only the the information you want in your report. Recall #| include: false and #| output: false

To do…

• Lab 4: Childcare Costs
  • Due Monday, 2/5 at 11:59pm
• Read Chapter 5: Special Data Types
  • Check-in 5.1 - 5.2 due Tuesday (2/6) at 8:00am